Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction

We consider the problem of off-policy evaluation (OPE) in reinforcement learning (RL), where goal is to estimate performance an policy, pie, using a fixed dataset, D, collected by one or more policies that may be different from pie. Current OPE algorithms produce poor estimates under policy distribution shift i.e., when probability particular state-action pair occurring pie very same D. In this work, we propose improve accuracy estimators projecting high-dimensional state-space into low-dimensional concepts state abstraction literature. Specifically, marginalized importance sampling (MIS) which compute correction ratios their estimate. original ground state-space, these have high variance lead OPE. However, prove lower-dimensional abstract can lower resulting then highlight challenges arise estimating data, identify sufficient conditions overcome issues, and present minimax optimization whose solution yields ratios. Finally, our empirical on difficult, tasks shows make MIS achieve mean-squared error robust hyperparameter tuning than